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Box No. I Basis of the opinion 

1. With regard to the language, this opinion has been established on the basis of: 
B the international application in the language in which it was filed 

□ a translation of the international application into , which is the language of a translation furnished for the 
purposes of international search (Rules 12.3(a) and 23.1 (b)). 

2. With regard to any nucleotide andbr amino acid sequence disclosed in the international application and 
necessary to the claimed invention, this opinion has been established on the basis of: 

a. type of material: 

□ a sequence listing 

□ tabfe(s) related to the sequence listing 

b. format of material: 

□ on paper 

□ in electronic form 

c. time of filingyfurnishing: 

□ contained in the international application as filed. 

□ filed together with the international application in electronic form. 

□ furnished subsequently to this Authority for the purposes of search. 

3. □ In addition, in the case that more than one version or copy of a sequence listing andbr table relating thereto 

has been filed or furnished, the required statements that the information in the subsequent or additional 
copies is identical to that in the application as filed or does not go beyond the application as filed, as 
appropriate, were furnished. 

4. Additional comments: 
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Box No. V Reasoned statement under Rule 43fc/s.1(a)(i) with regard to novelty, inventive step or 
industrial applicability; citations and explanations supporting such statement 

1. Statement 

Novelty (N) Yes: Claims 1-21 (but see Item V) 

No: Claims 

Inventive step (IS) Yes: Claims 

No: Claims 1-21 

Industrial applicability (IA) Yes: Claims 1-21 

No: Claims 



2. Citations and explanations 
see separate sheet 
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Re Item V 

Reasoned statement with regard to novelty, inventive step or industrial applicability; 
citations and explanations supporting such statement 

1 Reference is made to the following documents of the international search report: 

D1 : XP-2 397 141, Eggplant Version 3.1 Tutorial, 3 June 2005 

D2: XP-002388052, David Herron's Blog, "Accessibility for test automation" 

D3: US-A-5 335 342, (Tiburon Systems Inc.), 1994-08-02 

2 The following clarity problems have to be addressed first: 

2.1 Claims 1 ,8,1 5 have been drafted as separate independent claims apparently relating 
to very similar subject-matter but differing from each other widely with regard to the 
definition of the subject-matter for which protection is sought and, most notably, in 
respect of the terminology used for the features of that subject-matter. The 
aforementioned claims therefore lack clarity when read in their entirety. In order to 
overcome this objection it would be necessary to closely harmonise the wording of 
the independent claims. 

2.2 The term contour arises first in claims 5,1 2,1 9 and thus lacks antecedent support in 
claims 6,7,13,14,20,21 . The category of claim 21 is "system". 

2.3 The following expressions of claims 1 ,8,15 are unclear/vague/have no commonly 
recognised/unique technical meaning and thus have to be clarified in the claims 
based on the definitions given in the description: 

1 . "structural content filtration" - what is "structural contentTcontent" of what ? 

2. "active object" - what is "active" ? "object" of what ? is it the "object to be 
searched for" ? 

3. "recorded image generated by a graphical user interface" - what is the role of 
the GUI exactly ? what is the meaning of "recorded" here ? 

4. "playback image" - what is played back ? why is there an "image" ? 

5. "subset" - why "sub"-set ? (lack of antecedent support) 

6. what steps are automatic/what steps require user input ? 
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7. how do the claimed steps relate to the wanted automated control of the 
application software ? 

In order to further explicate the above objection the following exemplary wording of 
claims 1 ,8,15 - which would overcome the objection - is presented: (example for 
claim 1, corresponding examples for claims 8,15 implied): 

"An automated computer implemented method of structural content filtration to 
reduce the number of hypotheses for the location of an active object of a graphical 
user interface GUI in an image of said GUI in order to automatically control, based on 
user activity recorded during a recording phase, execution of the application program 
having said GUI during a playback phase, said method comprising: 

recording an image depicting GUI objects, including said active object, 
displayed by said application program during said interactive recording phase, 
transforming the recorded image and a corresponding playback image of said 
GUI captured during said playback phase; 

determining a sub-image from the recorded image which corresponds to said 
active object to be searched for in the transformed playback image; 
determining a set of points on the transformed playback image which have 
appropriate values for matching the sub-image; and 
filtering location hypotheses on the playback image which are more than a 
selected distance from any point in said set in order to identify the GUI object to 
be activated next to continue playback" 

2.4 Without the clarification in the last sentence the subject matter of claims 1 ,8,1 5 could 
also be misunderstood and accidentally read on comparing areas of the test screen 
shots to the playback screens hot areas in order to identify the differences due to 
software bugs (cf. D3, claim 1). 

2.5 For the following the subject matter of claims 1 ,8,1 5 will be interpreted as if already 
clarified. 

3 The claimed subject matter lacks an inventive step for the following reasons: 
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3.1 The claimed subject matter of claims 1 ,8 and 15 can be compared with the operation 
of a human operator performing manual testing sessions of developed computer 
applications. Indeed this is explicitly admitted on pg. 2 line 30-pg. 3 line 4. Claimed 
subject matter amounting to nothing more than the mere idea of automating human 
behaviour does not involve any inventive activity, in particular: 

3.2 Considering that previous research in the area of image recognition has been 
strongly inspired by human vision and that existing program analysis systems also 
already use image recognition techniques (see section "Description" on pg. 1 of the 
present application), it was an obvious aim to design a computer-implemented 
method/system for GUI testing by searching for an active GUI object by automating 
steps which a human being would perform in a similar situation (replacing the human 
tester by an equivalent machine). 

3.3 The closest prior art is taken to be the way a human would memorize and re-execute 
graphical user input in order to control a computer program. 

Assume, for example, that a person is taught a sequence of commands on one 
screen e.g. in a course or through a demo/tutorial, or upon being instructed on how to 
carry out a manual software test Most persons will visually memorize the appearance 
of the GUI and the appearance of the buttons which have to be clicked. When it 
comes to replay this sequence of events, e.g. at the own computer, the person will try 
to match the memorized pictures with the GUI as it currently appears in order to re- 
click the buttons in the right order. 

Note that the person's computer might well have a screen of different size/colour 
scheme etc. and thus the memorized picture and the currently seen picture would 
have to be mentally transformed to take account of these trivial differences in 
appearance. 

Note also that the person would mentally "determine a set of points", namely those 
points in the current screen image which match the appearance of the memorized 
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active button sub-image of the screens hot image. 

Finally, note that filtering hypotheses arises naturally and frequently in human and in 
machine recognition. For example, if the person should have partially forgotten which 
button was the right one during training then the overall layout and its distances 
constitute a common visual cue for hypotheses filtering. For example: "The button 
was in the upper left during training (recording) so I expect it to be there again and I 
wont click on one of the more distant buttons during actual working sessions 
(playback)". 

3.4 Thus, the features of claim 1 (and accordingly also those of claims 8 and 1 5) are 
merely those which a skilled person would make a machine perform in order to 
correspond to the way a human being would accomplish the same task of "extracting 11 
and re-finding "objects" within his field of vision. Thus, the claimed method amounts 
to nothing more than the mere idea of automating human behaviour and hence does 
not involve an inventive step (Article 56 EPC). 

3.5 The above analysis also extends to the subject matter of dependent claims 2-7,9- 

1 4,1 6-21 : it is a well documented fact of perception research that human beings also 
use the visual outline of objects when recognising them: hence the corresponding 
use of contours (edges) is trivial. So is the transforming into "black/white" as most 
human beings do not memorise the colours anyway, and, moreover, not disregarding 
color would evidently render the approach non robust w.r.t. changes in user selected 
color schemes. Finally, "black/white" renderings are most common during OCR which 
would also be a standard algorithm most usefully employed on the text parts of the 
subimages. 

LACK OF INVENTIVE STEP W.R.T. PRIOR ART DOCUMENTS: 

3.6 In the present case, it is also possible to alternatively argue as follows: 

3.7 Prior art document D1 ("eggplant™") teaches capturing GUI image areas and saving 
them as templates during recording. During playback these image areas are 
searched in the playback screen image, and the corresponding active objects of the 
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GUI are activated if a match between a region in the playback image and the 
recorded captured template can be found (see, for example, "The Finished Script" on 
pg 16). Alternatively, one could start from D2 ("post by robogeed on July 25, 2005 at 
7:38 AM") or also from D3 (claim 8 and col. 5 lines 25-51 ) as closest prior art. 

3.8 Hence, the closest prior art might be interpreted as differing from the subject matter 
of claims 1 ,8,15 basically in that it does not teach filtering of hypotheses on the 
playback image which are more than a selected distance from any of the subset of 
points. 

3.9 The technical problem would thus be: improve the template matching used in D1 . 

3.10 Recognising this problem does not constitute by itself an inventive step. Correct 
template matching being crucial for the success of D1 , it is evident that making it as 
robust as possible is highly desirable. 

3.1 1 The solution recited in claims 1 ,8,15 also does not involve an inventive step. Note 
that the close similarity of the task at hand (re-locating sub-regions of a previous 
image in a later subsequent image) to the general problems encountered in image 
registration and in tracking image regions is palpable. Adding a step of filtering 
among multiple hypotheses based on further cues is a standard technique in 
computer vision in general, and in particular so in tracking and in image registration. 
Moreover, distances are also commonly employed for that purpose, i.e. to eliminate 
wrong template matches in image registration and tracking. Hence, in the present 
context, using the distances among the objects of the GUI for hypothesis filtering 
constitutes an obvious solution that is obtained through entirely routine-reasoning 
within the framework of the normal progress of technology. 
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